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ABSTRACT 

Objective: To assess the discrimination 
and calibration of the Pediatric Index 
of Mortality 2 in patients admitted to a 
pediatric intensive care unit. 

Methods: The study was conducted 
with a contemporary cohort from 
November 2005 to November 2006. 
Patients aged 29 days to 18 years were 
included in the study. Patients who died 
within 12 hours of admission and cases 
of readmission were excluded from the 
study. The performance of the Pediatric 
Index of Mortality 2 was assessed 
by means of the Hosmer-Lemeshow 
goodness-of-fit test, the standardized 
mortality ratio and the area under 
receiver operating characteristic (ROC) 
curve with 95% confidence interval. The 
significance level was established as 5%. 

Results: A total of 276 admissions 
to the pediatric intensive care unit were 
included in the analysis. The mortality 
rate was 14.13%, and the efficiency 



of admission 0.88%. The median age 
of the sample was 42.22 months, and 
most participants were male (60.1%). 
Most admissions were referrals from 
the emergency department. The mean 
duration of stay in pediatric intensive care 
unit was 6.43±5.23 days. Approximately 
72.46% of admissions were for clinical 
reasons and exhibited an association 
with the outcome death {odds ratio: 2.9; 
95%CI: 1.09-7.74; p=0.017). Calibration 
of the Pediatric Index of Mortality 2 
with the chi-square statistic was 12.2686 
(p=0.1396) in the Hosmer-Lemeshow 
goodness-of-fit test, and the standardized 
mortality ratio was 1.0. The area under the 
ROC curve assessing model discrimination 
was 0.778. 

Conclusion: Pediatric Index of Mortality 
2 exhibited satisfactory performance. 

Keywords: Intensive care units, 
pediatric; Outcomes assessment (Health 
care); Mortality; Quality indicators, health 
care; Risk assessment; Child 
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Ever since their appearance in Sweden in 1955,'*' pediatric intensive care 
units (PICUs) have provided a relevant contribution to the care of severely ill 
children based on the best use of human resources and high-cost equipment as 
well as continuous monitoring of patients, thus allowing for early intervention 
and better therapeutic outcomes, including recovery and prevention or reduction 
of permanent disabling injuries. However, this modality of care is associated 
with high financial costs. In the United States, the expenses associated with 
intensive care represented approximately 1 % of the gross domestic product in 
1994, i.e., USD 67 billion.^* 
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The results of studies that investigated the demographic 
profile, mortality, morbidity and average stay in PICUs 
vary as a function of characteristics particular to various 
countries and even among different health services in the 
same country These divergences not with standing, there 
is a consensus about the need for studies to improve the 
understanding of pediatric intensive care outcomes to allow 
for more adequate allocation of human, technological and 
financial resources. 

For the reasons outlined above, studies have been 
performed to assess various methods to predict the severity 
and mortality risk of children, including the Pediatric Risk 
of Mortality (PRISM),») PRISM III,'* Pediatric Index of 
Mortality (PIM),<5> PIM2'^' and PIM3.'^' The aim of the 
present study was to investigate the performance of PIM2 
at the PICU of the Hospital Infantil Nossa Senhora da Gloria 
(HINSG) by analyzing its calibration and discrimination. 

METHODS 

HINSG is affiliated with the public hospital network 
established by the State Department of Health of Esptrito 
Santo, Brazil and is a state and regional reference hospital 
for specialized high-complexity pediatric care. The 
six-bed PICU of HINSG is a teaching and care-providing 
center that admits patients up to age 18 years, with 340 
admissions per year on average. 

Prospective data collection was performed from 
November 2005 to November 2006 in a contemporary 
cohort comprising all patients aged 29 days to 18 years. 
Patients who died within 12 hours of admission were 
excluded from the study, as well as cases of readmission and 
infants aged zero to 28 days. The study was approved by 
the Research Ethics Committee of HINSG, no. 43/2005, 
with a waiver of informed consent. 

Efficiency of admission was defined as the percentage 
of patients that exhibited PIM2 >1% at admission. 
PIM2'"'' includes clinical and laboratory data that are 
firmly established as part of the intensive care routine 
within the first hour of admission (blood pressure, use 
of mechanical ventilation, pupillary light reflex, arterial 
blood gases, recovery from surgical procedures, high- or 
low-risk diagnosis and elective admission or not). Thus, 
no intervention was performed exclusively for research 
purposes or involved risk for the patients. The PIM2 
scoring system is in the public domain, and its authors 
authorized its use in the present study. 

The performance of the PIM2 was assessed through 
analysis of its calibration and discrimination. Calibration 
evaluates how well the model classifies patients into low-. 



medium- and high-risk categories. For that purpose, the 
Hosmer-Lemeshow goodness-of-fit test was used, which 
is based on individual PIM2 values, and displays the 
observed-expected mortality ratios in deciles or in standard 
risk categories (<1%, 1-4%, 5-15%, 1629%, >30%). For 
this test, p-values >0.05 denote a good fit. The expected 
total mortality is the sum of the probabilities of death in 
each category. The standardized mortality ratio (SMR) 
represents the ratio of observed to expected mortality 
rates; values <1 imply good performance."*' 

Model discrimination was inferred from the area 
under the receiver operating characteristic (ROC) curve 
plotted using PIM2 values. The ROC curve represents 
the relationship between the sensitivity and specificity of 
a given test''' and in the present case expresses how well 
the model distinguishes between patients who lived and 
those who died. Although for some authors an area under 
the ROC curve of 0.75 or more is considered clinically 
useful,***' for others, 0.70 is adequate. ''"' 

Analysis was performed using Statistical Package for the 
Social Sciences (SPSS) software version 11.0, Epilnfo™ 
version 3.3.2 and MedCalc version 9.2.1.0. Descriptive 
statistics were performed (frequency tables, calculation 
of means and standard deviation), as well as exploratory 
hypothesis testing and application of the chi-square test of 
association. The significance level was established as 5%. 

To analyze the performance of the PIM2, the 
Hosmer-Lemeshow goodness-of-fit test was used,"" and 
the SMR and area under the ROC curve were calculated"^' 
together with the corresponding 95% confidence 
intervals (95%CI). 

RESULTS 

A total of 333 patients were admitted to the 
HINSG-PICU during the study period, and 57 were 
excluded (30 were newborn infants, 18 were readmissions, 
six died within 12 hours of admission, two were transferred 
to other hospitals and one was older than 18 years). 

Relative to the 276 patients included in the study, the 
mortality rate was 14.13%. The efficiency of admission 
was 88%, the median age of the sample was 42.2 months, 
and most participants were male (60.1% of admissions). 
None of these demographic variables exhibited statistically 
significant correlation with an outcome of death (Table 1). 
With regard to patient origin, 48.9% were referred by the 
hospital emergency department, and this group exhibited 
a significant difference in death outcomes compared to the 
remainder of the sample (p=0.03). The median stay in the 
PICU was five days, with an average of 6.43±5.23 days. 
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Table 2 describes the interquartile distribution of the 
studied sample according to age (in months), duration of 
stay in the PICU (in days) and PIM2 scores. Admissions for 
clinical reasons corresponded to 72.4% of the total number 
and exhibited association with an outcome of death (odds 
ratio- OR: 2.9; 95%CI: 1.09-7.74; p=0.01) (Table 3). 

Table 1 - Overall characteristics of tine sample 



Variables 


Results 


Number of patients/deaths 


276/39 


Mortality rate 


14.13 


Efficiency of admission 


88 


Age Imonthsl 


42.22 (5.38-122.25) 


Origin 




Emergency department 


48.91 


Hospital wards 


25.00 


Surgery department 


17.03 


Other hospitals 


9.06 


Duration of stay (days) 


5 (3-8) 


Reasons for admission 




Clinical 


72.46 


Surgical 


27.54 


Efficiency of admission: number of patients with Pediatric Index of Mortality 2 >1% at 
admission. Results expressed as number (%), mean±standard deviation, or median (25%-75%t. 


Table 2 - Interquartile distribution of the variables of age, duration of stay and 
Pediatric Index of Mortality 2 score (N = 276) 


Age in months 


Duration of stay in days PIM2 


Percentiles 




25 5.38 


3.00 1.6892 


50 42.22 


5.00 4.8373 


75 122.25 


8.00 10.1550 



PIM 2 - Pediatric Index of Mortality 2. 

Table 3 - The variables origin and reason for admission according to outcomes 

Outcomes 



Variables 

Discharge Death 

Origin 

Emergency department 1 1 7 (49.37) 1 8 (46.1 5) 

Hospital wards 57(24.05) 12(30.77) 

Surgery department 45(18.99) 2(5.13) 

Other hospitals 18(7.59) 7(17.95) 
Admission 

Clinical 166(70.04) 34(87.18) 

Surgical 71(29.96) 5(12.82) 

Total 237(100) 39(100) 

Results expressed as number (%). 



p value 



0.03 



0.01 



Tables 4 and 5 describe the results of the 
Hosmer-Lemeshow goodness-of-fit test and the SMR 
values expressed in deciles according to the five standard 
mortality risk categories described in the literature. The 
chi-square statistic was 12.26 (p=0.13) across deciles 
and 1.34 (p=0.71) across the five risk categories. In both 
cases, the overall SMR was 1.0, although the variation 
corresponding to the highest and lowest risk scores 
was overestimated. Figure 1 depicts the comparison 
between the expected and observed mortality rates per 
risk category. The area under the ROC curve was 0.77 
(95%CI: 0.720.82), as shown in figure 2. 



Table 4 - Hosmer-Lemeshow goodness-of-fit test across Pediatric Index of 
Mortality 2 deciles and standardized mortality ratio according to the observed 
and expected mortality rates 



Outcomes 



Survival 



Death 



PIIVI2 


Observed 


Expected 


Observed 


Expected 


SMR 


1 


28 


26.112 


0 


1.888 


0 


2 


27 


26.028 


1 


1.972 


0.51 


3 


27 


25.950 


1 


2.05 


0.49 


4 


26 


25.781 


2 


2.219 


0.90 


5 


25 


25.602 


3 


2.398 


1.25 


6 


25 


25.399 


3 


2.601 


1.15 


7 


23 


25.137 


5 


2.863 


1.75 


8 


25 


24.390 


3 


3.61 


0.83 


9 


17 


21.934 


11 


6.066 


1.81 


10 


14 


10.668 


10 


13.332 


0.75 


Total 


237 


237.001 


39 


38.999 


1.00 


PIM 2- Pediatric Index of Mortality 2; SMR- 


standardized mortality ratio. 


12.26; p=0.13. 


Table 5 - Hosmer-Lemeshow goodness-of-fit test across Pediatric Index of 
Mortality 2 risk categories and standardized mortality ratio according to the 
observed and expected mortality rates 






Outcomes 








Survival 






Death 




PIIVI2 


Observed 


Expected 


Observed 


Expected 


SMR 


0 1-1 


33 


32.22 


0 


0.78 


0 


1 1-5 


102 


102.54 


7 


6.46 


1.08 


5 1-15 


72 


72.11 


12 


11.89 


1.01 


15 1-30 


19 


20.26 


10 


8.74 


1.14 


>30 


11 


9.87 


10 


11.14 


0.90 


Total 


237 


237.00 


39 


39.01 


1.00 


PIM 2 - Pediatric Index of Mortality 2; SMR - 


■ standardized mortality ratio. 


= 1.34; p=0.71. 



Rev BrasTer Intensiva. 2014;26{l):44-50 



47 Netto AL, Muniz VM, Zandonade E, Maciel EL, Bortolozzo RN, Costa NR Limongi RS 



35% 
30% 
25% 
20% 
15% 
10% 

5% 

0% 

0-1% 1-5% 5-15% 15-30% >30% 
Figure 1 - Deaths according to risl< categories - Pediatric Index of IVlortality 2. 



= = = Observed deaths 
— - = Expected deaths 




0.4 

1 - Specificity 

Figure 2 - Area under the ROC curve of the Pediatric Index of IVlortality 2; 0.77 
(95% confidence interval: 0.72-0.82). 



DISCUSSION 

The results of the present study indicate that the 
performance of the PIM2 was good. In addition, an 
association was found between mortality and admission 
for clinical reasons, as well as with origin from the 
emergency department. The duration of stay in the 
PICU was similar to those reported in other studies, 
with a mortality rate of 14.13% in the present study. 
The present study is part of a master dissertation project 
investigating the epidemiology of a PICU, in which the 
nosological features of the patient population and health 
care resources remained stable after the end of the study. 
Therefore, it may be concluded that the investigated 
PICU complies with the current recommendations of the 



Brazilian Health Surveillance Agency {Agenda Nacionalde 
Vigilancia Sanitaria - ANVISA) for the care of severely 
ill children. The investigated PICU does not perform 
therapeutic interventions, such as the use of extracorporeal 
membrane oxygenation (ECMO), inhaled nitric oxide, 
high frequency oscillatory ventilation, transplantations 
and use of the Swan-Ganz catheter. 

The actual availability of PICU beds in our milieu 
results in a high prevalence of admissions of very 
severely ill individuals. In addition, as our hospital is a 
regional reference hospital for pediatric subspecialties, 
approximately 30% of the patients admitted to PICU 
exhibit comorbidities. 

These facts not with standing, the healthcare outcomes 
based on the PIM2 score are similar to those reported 
by other centers for the same type of patients. Studies 
conducted in countries with limited resources found a 
mortality rate of 32%,*'^* while Briassoulis'*'" reported a 
mortality rate of 12% in one Greek PICU, and Brady''^' 
reported mortality rates varying from 3.2% tolO.1% in 
various PICUs in the United Kingdom. 

Of the demographic variables, age has been associated 
with an outcome of death. Einloft et al.*"'' found a 
mortality rate of 13.22% among infants younger than one 
year, with a relative risk (RR) of 1.86 (95%CI: 1.65-2.10; 
p<0.0001). De Freitas Aragao et al."^' found a statistically 
significant association between death in children admitted 
to the PICU and age below two years (p=0.007). In the 
present study, however, no significant differences were 
found in this regard. 

A study conducted by El-Nawawy*'*' that included 
406 children admitted to the PICU of a pediatric teaching 
hospital in Egypt over a period of 13 months found a 
mortality rate of 38%. Approximately 57.9% of the 
children admitted to that PICU had been referred from 
the emergency department, although with no significant 
difference in outcome. In this context, the features 
inherent to emergency care of severely ill patients should 
also be taken into consideration, as interventions are 
performed in settings and under conditions quite different 
from those in other hospital departments. '''' This situation 
entails several possible determinants, including the type of 
care afforded, issues related to access to healthcare services, 
particularly high-complexity issues, organization of the 
healthcare network and difficulties associated with the 
regional distribution of specialized care. Therefore, further 
studies assessing those features are needed. 

The association found in the present study between 
admissions for clinical reasons and an outcome of death agrees 
with findings reported by other authors. In a study conducted 
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by Lopez-Herce et al.,'^°' 66.5% of admissions were for 
clinical reasons and associated with a greater mortality rate 
compared to surgical patients, 6.5+3.7% versus 2.6±3.2%. 

The duration of stay of severely ill patients in the ICU 
varies among the various centers and ranges from 6.8 
to 11.6 days on average. In the present study, the 
duration of stay of approximately 43% of non-survivors 
was four to seven days, and approximately 70% of deaths 
occurred within seven days of admission. 

In their study conducted in Greece, Briassoulis et al.*'* 
found an increase in the duration of stay in the PICU in 
2001 following a period of stability (1997-2000), which 
they attributed to an increase in the number of admissions 
of patients with chronic diseases. In contrast, the 
case-control study conducted by van der Heide et al.*^^' did 
not find significant differences between the characteristics 
of long-term patients and the control group. 

The Hosmer-Lemeshow goodness-of-fit test yielded a 
chi-square value of 12.26 and p=0.l4 across the analysis 
in deciles and a chi-square value of 1.34 and p=0.71 across 
the five risk categories. The overall SMRwas 1.0, although 
the variation corresponding to the strata with the highest 
and lowest risk scores was overestimated. These results 
indicate satisfactory calibration. 

In a study by Thukral et al . , " ^' the overall SMR was 1.57, 
and with the Hosmer-Lemeshow goodness-of-fit test, the 
chi-square value was 7.64 (p=0.47). These authors called 
attention to the fact that their study population from India 
exhibited demographic and socioeconomic profiles that 
were different from those in the country where PIM2 was 
first formulated, in addition to a greater load of severity of 
illness being managed with fewer resources and differences 
in the quality of care, which may have contributed to an 
underestimation of the mortality risk. 

Although the SMR was 0.85 in a study by Eulmesekian 
et al.'^'", the Hosmer-Lemeshow goodness-of-fit test did not 
yield significant results. These authors observed that several 
researchers have raised concerns about the use of the PIM2. 
In fact, the original PIM2 formulators recommended 
caution in the use of the Hosmer-Lemeshow goodness-of-fit 
test because small and clinically irrelevant differences within 
a large sample could exhibit the same p-value as large and 
clinically relevant differences within a small sample. 

One further topic deserving of attention concerns the 
collection of data at admission. At that time, as a function 
of the support measures established, some children may 
exhibit stable clinical conditions despite the severity of 
disease, the progression of which over time may thus 
contradict predictions associated with low PIM2 scores, as 
Martha et al.'^^* discussed with regard to the PIM. 



Slater et al.""' proposed that the variations in PIM2 
calibration maybe associated with the following factors: 
results of patient intervention, changing attitudes as 
to the indications for commencing and discontinuing 
life support, changes in the thresholds for admission to 
intensive care, and the clinical profile of patients. These 
factors not with standing, these authors indicate a cutoff 
point of 1.0 for the PIM2 SMR. 

The area under the ROC curve found by various authors 
indicates that the discrimination of the PIM2 is adequate: 
0.90 (95%CI=0.89-0.92);<2^) 0.81 (95%CI=0.750.87);'"' 
0.84 (95%CI=0.82-0.86);<'5> 0.97 (95%CI=0.96-0.99).<'5' 
The variation among these values is due to the above 
mentioned impact of the profile of each individual center 
on model performance. In the present study, the area under 
the ROC curve was 0.778 (95%CI=0.725-0.826), which 
indicates that the model discrimination was satisfactory in 
the assessed population. 

The PIM2 exhibited mixed results in populations that 
were different from the one in which it was originally tested. 
For that reason in 2013, Straney et al. published an updated 
version of the PIM2 known as the PIM3,'^' which was based 
on an international, multicenter, prospective cohort study 
conducted in 2010 and 2011. That version formulated 
a novel stratification of the variables corresponding to 
recovery from surgical procedures (cardiac surgery with 
bypass, cardiac surgery without bypass and other surgical 
procedures) as well as the risk categories, which are currently 
defined as low-, high- and very high-risk. The model 
discriminatory performance was better in Australia and New 
Zealand (area under the ROC curve: 0.91 [0.90-0.93]), 
compared to the United Kingdom and Ireland (area under 
the ROC curve: 0.85 [0.84-0.86]). 

According to the current evidence, the predictive 
ability of the PIM2 is not affected when it is calculated 
using data collected four hours after admission to the 
PICU.<^^' Other authors found that the PIM2 score 
discriminated well between survivors and deaths in the 
PICU and emphasized the use of PIM2 in the stratification 
of interventions.'^^* One study conducted in a mixed ICU 
raised some concerns as to the use of the PIM2, as there 
was overestimation of deaths in the highest risk group, 
more particularly in cardiac-surgical patients.*^** This 
adjustment was based on the PIM3.'^' 

The mortality risk prediction indices should be 
considered as auxiliary tools for the management of 
healthcare services and critically assessed as a function of 
each situation, and they must be systematically applied 
to the analysis of groups of patients and should never 
influence the management of individual cases. The 
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present study had some limitations. As a function of its 
cross-sectional and descriptive design, the results entail 
limitations for the investigation of phenomena. Then, 
because the study was conducted in a single PICU, the 
results may not be extrapolated to other health services 
with different characteristics. Finally, the data collected for 
analysis are quite old, although the epidemiological and 
structural profile of the investigated PICU did not exhibit 
any changes, but the results agree with those reported in 
more recent studies. 



CONCLUSION 

The present study showed that the discrimination 
and calibration of the Pediatric Index of Mortality 2 
were adequate. It further found an association between 
mortality and admission for clinical reasons, as well 
as with origin from the emergency department. As a 
function of its operational characteristics, the Pediatric 
Index of Mortality 2 may be used as an auxiliary tool for 
the management of intensive care units. 



RESUMO 

Objetivo: Avaliar a discriminafao e a calibra^ao do Pediatric 
Index of Mortality 2 em pacientes de uma unidade de cuidados 
intensivos pediatrica. 

Metodos: Estudo de coorte contemporanea realizado no 
periodo de novembro de 2005 a novembro de 2006. Os limites 
de idade foram 29 dias de vida e 18 anos. Exclui'dos aqueles com 
obito em menos de 12 horas apos admissao e readmissoes. Para o 
desempenho do Pediatric Index of Mortality 2, foram aplicados o 
teste de Hosmer-Lemeshow, o indice padronizado de mortalidade 
standardized mortality ratio e area sob a curva ROC, intervalo de 
confianfa de 95%. O nivel de significancia foi de 5%. 

Resultados: Foram estudadas 276 admissoes de pacientes. A 
taxa de mortalidade foi de 14,13%, com eficiencia de admissao 
de 0,88%. A mediana de idade foi de 42,22 meses, havendo 



predominio do genero masculino com 60,1%. O setor de 
emergencia foi responsavel por 48,91% das admissoes. Tempo de 
permanencia foi de 6,43±5,23 dias (media). As admissoes clinicas 
corresponderam a 72,46% e associaram-se ao obito {odds ratio: 2,9; 
intervalo de confian^a de 95%: 1,09-7,74; p=0,017). O Pediatric 
Index of Mortality 2 apresentou calibra^ao com qui-quadrado de 
12,2686 (p=0,1396) no teste de Hosmer-Lemeshow, e indice 
padronizado de morte de 1,0. A discrimina^ao relacionada a area 
ababco da curva ROC foi de 0,778. 

Conclusao: O escore Pediatric Index of Mortality 2 apresentou 
desempenho satisfatorio. 

Descritores: Unidades de terapia intensiva pediatrica; 
Avalia^ao de resultados (Cuidados de saiide); Mortalidade; 
Indicadores de qualidade em assistencia a saiide; Medi^ao de 
risco; Crian^a 
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